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1. Introduction 

The long term objective of this research is to develop a system for remote robotic surgery which 
will permit surgery between any two places on earth with a patient in one location and the 
surgeon in another. In fact this surgery could also be performed with a patient on-board a 
spacecraft. The precise distance from earth, over which our approach is practical has not yet been 
determined. The major impediment to remote surgery is the effect of telecommunications delay 
on the surgeon’s performance. It has been shown in a myriad of studies of human in the loop 
systems that system delays lead to degraded operator performance and ultimately unstable 
systems. To quote Dr. Richard Satava, “During my development of the initial telesurgery systems 
through the Advanced Biomedical Technology program at DARPA, there was principal focus on 
the systems integration but the program was not able to resolve the latency issue This is an area 
which has had much speculation but little hard data, resulting in the off-hand dismissal of very 
remote telesurgery. ” [1] 

Since the delay cannot be eliminated, in order to accomplish this objective, the only solution is to 
mitigate the effects of the delay on the surgeon performing the operation. We have developed an 
intelligent systems approach, which is to have the surgeon operate through a simulator running in 
real-time. The use of a simulator enables the surgeon to operate in a virtual environment free 
from the impediments of telecommunication delay. The simulator functions as a predictor and 
periodically the simulator state is corrected with “truth” data (Won Soo Kim pred display ref). 

Several aspects of our approach will make use of a variety of forms of intelligent systems as will 
be explained below. The goal of mitigating the effects of time-delay in a practical way are so 
challenging that it can be realized only by making the best use of recently developed approaches 
to machine intelligence. 
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It is interesting to note that in the late 1980s, after its inception the utilization of laparoscopic 
cholecystectomy grew rapidly. However, minimally invasive surgery (MIS) for other operations 
has not experienced the same pattern of growth. According to Ballantyne [2], the reason is that in 
general laparoscopic procedures are hard to learn, perform and master. This is a consequence of 
the fact that the camera platform is unstable, the instruments have a restrictive number of degrees 
of freedom and the imagery presented to the surgeon does not offer sufficient depth information. 
The solution seems to be at hand with the significant growth of robotic surgery. This is surgery 
where-in the surgeon operates through a robot. In a sense this robot is a telemanipulator under 
the control of the surgeon. The robotic system provides a stable video platform, added dexterity 
and in some cases a stereoscopic view of the surgical field. 

Since proximal robotic surgery seems to be maturing the next logical step in surgical care is to 
extend to remote applications of robotic surgery. That is to say, the surgeon and the operating 
console are at one location and the robot and patient at another. The idea of remote robotic 
surgery, or as some refer to it, telesurgery, has been an objective for some time, especially in the 
military. This advancement is seen by the military as the means by which the next major 
improvement in battlefield survivability [3]. In addition to the military application, the 
technology could be useful if an astronaut were to require emergency surgery while on the space 
station. Furthermore, perhaps the most ubiquitous application will be in civilian medicine. 
Patients in medically remote areas would have the option of receiving an operation performed by 
a renowned surgeon even though the surgeon and patient may be thousands of miles apart. 


2. The Time Delay Problem in Telerobotic Surgery 

The main impediment to the availability of this technology is the communications delay 
associated with long distance signal transmission. This delay is inevitable and a consequence of 
the propagation speed of electromagnetic radiation. Figure 1 illustrates the signal paths. 
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Figure 1. Signal paths. 

It is well known that system delays will cause a deterioration of the human-machine system 
performance. As a matter of fact this is true for any control system, not only a human-in-the-loop 
control system. Figure 2 illustrates the time domain effect of delays of 0, 200, 400, and 800 ms 
where the input is a unit step. The graph indicates that as the delay increases, the response lags 
the input by a greater amount. In addition the 400 ms delay case seems to display limited 
stability, while the 800 ms delay case clearly exhibits unstable response. The system analyzed 
includes a fourth order plant and a human operator model, to which the delays are added. 

Figure 3 presents the results of frequency domain analysis of the same system. Here, one 
observes that the 400 ms delay case yields a phase margin of approximately zero, while the 
800 ms case has a negative phase margin. We can then examine human operator performance 
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data in a system with and without delays. There are many such examples in the literature. It is 
observed that when delays become long, human operators will adopt a move and wait strategy. 
This allows the operator to observe the results of his/her action before committing to another 
action. The move and wait strategy may be acceptable for controlling a lunar or Mars rover but it 
is unacceptable in tightly closed loop applications, robotic surgery being one of them. 


Closedloop Response to Unit Step Input (for different delays) 



Figure 2. Time domain effect of delays. 


Frequency Response with/without Time Delays 



Figure 3. Results of frequency domain analysis of the system. 
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When the delays in an aircraft flight control system become too long the control loop becomes 
unstable and the aircraft is said to display pilot induced oscillation (PIO). This is another case 
where the move and wait strategy will not work. Figure 4 illustrates the effect of delay on a 
system operator performing a tracking task with and without force feedback. There were several 
cases of delay (0, 80, 200 and 300 ms) in the force feedback. In all cases the subjects had a 
narrow field of view visual presentation. The graph shows that at 200 and 300 ms delay in the 
force feedback the operator’s performance is essentially as bad or worse as with no force 
feedback [4]. Whereas, at a delay of 80 ms his/her performance is much improved and almost as 
good as a fully synchronous feedback. 
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Figure 4. Effect of delay on system operator performance. 


3. Preliminary Studies to Determine Maximum Tolerable Delay 

Preliminary studies were conducted using experienced laparoscopists in a suture knot tying task. 
The task was performed using a laparoscopic training device with delays introduced, in 25 ms 
intervals, into the video monitor via an analog delay device from Prime Image. The performance 
metric used in this study was the time it took the subjects to complete the knot. Figure 5 
illustrates the results. For delays up to about 100 ms the execution time remained relatively 
constant at about 13 seconds. Above the 100 ms point the time increases substantially. 
Preliminary results seem to indicate that by the time the transport delay approaches 500 ms the 
time to complete the knot is about 90 seconds. One of the interesting results is that subjects 
began to experience nausea at delays approaching one second. This result was quite unexpected 
based on our considerable experience with simulator sickness. 
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Subject #1: Knot Tie Time as a Function of Delay Time 



Figure 5. [5] Knot tie time as a function of delay time. 


4. The Intelligent Systems Approach 

Although universally accepted definitions for such terms as “intelligent system” and “artificial 
intelligence” may never be found, it is nonetheless useful to start with a reasonable working 
definition of what we mean by an intelligent system as the term is used in this paper. George 
Klir, the founding director of the Center for Intelligent Systems at SUNY-Binghamton, states 
that “Intelligent systems are human-made systems that are capable of achieving complex tasks in 
a human-like, intelligent way.” [6] Restating this slightly, we consider a system to be intelligent 
if it is capable of a behavior that would be described by a typical observer as an intelligent 
behavior if a human were to perform it. 

Starting from this working definition, we feel that at least four aspects of our scheme, a 
simplified view of which is shown in Figure 6. for mitigating the time-delay problem in 
telerobotic surgery are clearly intelligent. First, at least three major components of the approach 
are intelligent systems when viewed independently. These are the simulator (particularly in its 
role as a predictor), the image understanding component, and the intelligent controller. 
Furthermore, the interaction and coordination of all components in the overall integrated system 
is a complex process that we view as the fourth aspect of intelligence. 



Surgical Telerobotics as an Intelligent Machine 

Figure 6. Surgical telerobotics as an intelligent machine. 
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5. The Simulator as Predictor 


Modern simulators tend to be very complex systems in their own right, but one particular aspect 
of how the simulator will function in this case is where we place the emphasis in calling our 
simulator an intelligent system. That is, in order for our time-delay-mitigation scheme to work, 
the simulator must predict what will happen in the surgical field before it happens. Furthermore, 
the predictive mechanisms in this case are based on dynamic modeling of a far from trivial sort. 
Adding still more complexity to the task, the system must be designed to allow the models to be 
updated in real time as the delayed information from the surgical field becomes available. 

Clearly, dynamics models both for the robot dynamics and organ dynamics are necessary for the 
simulator to function in this way. Though both are challenging, the organ dynamics modeling is 
known in medical research circles to be extraordinarily difficult, particularly in the case of soft 
tissue. For this, we intend to experiment with a variety of approaches, which will include both 
finite element analysis and continuum analytical models. 

The simulator as used here is clearly a perfect example of an anticipatory system. It is interesting 
to note that anticipatory behavior is often viewed in the literature [6, 7] as a primary 
characteristic in intelligent systems. 

6. Image Understanding 

An image understanding module on the patient side of the communications link (directly 
connected to the robot) is essential to the functioning of the overall system. The purpose of this 
component is to recognize the organs and various other objects in the surgical field from the 
video imagery coming from the surgical camera. It will feed this information on the location and 
states of these objects both to the intelligent controller (on the patient side) and to the simulator 
(on the surgeon side). To perform this image understanding task to the level of sophistication 
required in this research clearly fits any reasonable definition of an intelligent system. 

Developing the image understanding module will be a challenging part of the research. One 
aspect of its design is likely to be an image library. This library will consist of both generic 
(general anatomical information) and patient- specific (such as from MRI scans) information on 
the surgical field against which the imagery from the camera will be compared. One aspect of the 
approach that will facilitate the interpretation of the imagery will be to pick out easily 
identifiable landmarks as navigational aids. 

However, even using the image library and the landmarks, the task of interpreting the input from 
the camera in real time will still be a very challenging one. Some of the most promising new 
approaches for real-time image recognition are based on machine learning. In particular, they 
make use of a fast and very powerful approach called support vector machines, or what are more 
generically known as kernel-based methods. The power of these methods is that a simple data 
transformation followed by a linear model effectively constitutes a powerful nonlinear model. 

Image understanding encompasses the processing, encoding, and recognition tasks that will be 
accomplished on the patient side, using the video from the surgical camera(s). The problem of 
image identification has been a focus of research in the community of image understanding for a 
long time, and still there is no satisfactory solution available in general. However, when the 
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problem is more constrained into a specific application domain with a specific application 
scenario, there are feasible and robust solutions well that are well suited. 

A new approach to object recognition and categorization in image is under development by 
members of our team. A camera image is basically a 2D projection of a 3D world in the field of 
view of the camera. The approach is to reverse this process by determining the 3D world that 
generated the image. This approach has been successful in extracting 3D objects, including 
buildings and trees, from 2D images. The software to do this in real time has been developed to 
enhance 2D images, detect specified objects, extract 3D objects, and remove others from the 2D 
image. Although buildings are generally rectilinear, the objects extracted can be of an arbitrary 
shape. The extraction process is done in two steps. First, the object is detected or recognized. 
Second, the 3D parameters are extracted. 

The method being proposed is geometry-based in that it will use a library of 3D models that can 
describe what is being seen in the camera (organs, veins, etc.). These models are not rigid 
models, but rather are models that can change shape depending upon specified parameters. These 
models are likely to be patient specific, having been generated by an offline process prior to the 
surgery. 

The method decomposes the 2D image into regions. Regions are contiguous sets of pixels that 
are determined to belong to a specified set based on any arbitrary criteria. Examples of these 
criteria are color or texture. All objects in the 2D scene are composed of one or more of these 
regions. The regions in the image are then analyzed to determine the boundaries of the 2D 
projected objects, with the result being input to algorithms for determining the depth component, 
defined as the axis normal to the plane of the camera. This component for any surface is 
determined using calculations based on color, texture, and diffuse and specular reflection from 
the surface. Some objects can be obscured by other objects. Algorithms are required to determine 
when this is occurring. 

The edges of the 2D objects are found with pattern searches and correlated with the library 
model data to ascertain the actual 3D structure at the time the 2D image was obtained. The 
accuracy of the resulting 3D model will depend upon a number of factors, including the 
resolution of the camera, whether the image is in color or not, the quality of illumination, and the 
availability of accurate models of the objects being displayed. 

7. The Intelligent Controller 

As its name implies, we certainly consider the intelligent controller to be an intelligent system, 
even when viewed as an independent unit. This device, located on the robot/patient side of the 
communication link, performs in two critical roles. In the ultimate system for use on actual 
patients, the intelligent controller will be necessary to provide both an added measure of safety 
and an improved level of efficiency in the presence of time delay. Both the safety role and the 
efficiency-enhancement role require intelligent behavior. 

The need for an added element of safety in the presence of time delay is quite obvious. For a 
variety of reasons, even when the surgeon as well as the various other components of the system 
are performing perfectly, the existence of time delay prevents the possibility of 100% certainty 
as to where various tissue will be in relation to the surgical instruments at any given instant in the 
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future. Because the intelligent controller will be proximate to (and linked directly to) the robot, it 
will interact with the robot without significant delay, and thereby has the potential to control all 
robot movements instantaneously. Thus, as a last line of defense against the possibility of 
accidental collisions between surgical instruments and the patient’s vital organs, the intelligent 
controller will ultimately play a critical role. 

The need for improving the level of efficiency over what it would otherwise be in the presence of 
time delay is also clear. Finishing surgery in a timely manner and preventing unnecessary 
frustration for the surgeon are always important goals. While it may be true that the time delays 
associated with telerobotic surgery will never allow it to be quite as efficient as proximate 
robotic surgery, the goal at least must be to make it ultimately as efficient as possible. 

Although in the course of the research, we will attempt to apply a variety of advanced 
approaches to machine intelligence in designing effective intelligent controller prototypes, some 
basic aspects need not be particularly complex. For the safety aspect, a fairly effective controller 
could be based on nothing more than a three-dimensional geometric model of the surgical field 
combined with a simple type of production rule system. A typical rule for the case of gall bladder 
surgery might look roughly like the following: 

IF left end-effector holds sharp instrument AND 

instrument is within 5 mm of common bile duct AND 
an override command has not just been submitted by the surgeon 

THEN stop movement of left end-effector immediately, 
send safety alarm signal to surgeon, 
wait for reset by surgeon 

The production rule system for the case of gall bladder surgery may be comprised of perhaps a 
few dozen such rules. This is a very basic form of the traditional approach to artificial 
intelligence. Using this as a starting point, we can readily add more sophisticated machine 
intelligence approaches. 

One fairly straightforward addition would be the use of fuzzy logic in the production rules. 

Fuzzy logic is simply a calculus for representing mathematically the way humans use somewhat 
vague concepts, such as “very close” or “rapidly”, when reasoning about complex systems. For 
example the rule above could be made more sophisticated by changing the antecedent to the 
following: 

IF left end-effector holds sharp instrument AND 

(instrument is very close to common bile duct OR 
[instrument is somewhat close to common bile duct AND 
left end-effector is moving rapidly]) AND 
an override command has not just been submitted by the surgeon 

Naturally, when using fuzzy logic, it will also be necessary to represent in the system those 
numerical values associated with such terms as “very close,” “somewhat close,” and “rapidly.” It 
is quite simple to represent such terms in a particular context for relative locations, velocities, 
accelerations, and any other types of variables we may use. 
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Fuzzy logic is one component of what is now known widely as “the soft computing approach.”. 
Soft computing (SC) is a term coined by Lotfi Zadeh around 1990 to represent the emerging 
trend to design complex systems based on hybrids of four component methodologies, each of 
which had been evolving over the previous three or four decades. These component 
methodologies of SC are referred to most generically as fuzzy logic (FL), artificial neural 
networks (ANN), evolutionary computing (EC), and probabilistic reasoning (PR). The key 
concept of this approach is not just that these four methodologies tend to be powerful in and of 
themselves, but rather that there tends to be a synergistic effect when two or more of them are 
combined in appropriate hybrids. The SC approach, also referred to as computational 
intelligence, may seem to the layman a bit like science fiction, but it is a successful and well 
established amalgam of methodologies in some fields of engineering and is based ultimately on 
decades of advanced research. 

The success of SC has been demonstrated most graphically in the context of feedback control, 
particularly in inherently complex control applications. There have been very many citations in 
the literature of the successful application of SC hybrids, including for example in Lewis [8]. 
Certainly we will experiment with applying them here as well, and we can already state with 
confidence that they will be useful in the context of the intelligent controller. 

We will also experiment with another approach known in general as optimal control techniques, 
which use a model reference approach. In this case a model of the entire system; patient, robot 
and surgeon is employed along with a cost function which will be minimized to determine the 
coefficients of the parameters in the control laws. This need not be viewed as an entirely separate 
approach in the sense that optimal control concepts are often part of the SC methodology as well. 


8. The Integrated Intelligent System 

The fourth intelligent element is actually the total integrated system. The total system behaves as 
the human surgeon would if there were not a performance encumbering delay. Because the 
simulator through which the surgeon operates is running in real time the surgeon sees reaction to 
inceptor movements much more quickly than would be the case if he/she were required to wait 
while the signals made a complete round trip over the long haul network. Since, in robotic 
surgery, the surgeon is already in a synthetic environment the introduction of a simulator does 
not significantly alter the physician’s perceptual stimuli The operating station containing the 
control inceptors and the visual displays is the same as that used to control the surgical robot in 
the conventional configuration. In fact because of the addition of haptic stimuli the surgeon’s 
environment will be more stimulating. 

As previously stated in our embodiment the simulator acts as a predictor, providing information 
to the surgeon consistent with the no delay situation. The image understanding portion is the 
essential corrector. The intelligent controller is designed as an optimizer. Figure 7 is a detailed 
representation of the proposed system and the general research areas. The following paragraphs 
explain the architecture. 
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Figure 7. Detailed representation of the proposed system and general research areas. 

The signal path from the surgeon’s inceptor movement proceeds to the simulator and 
simultaneously to the intelligent controller, which commands the robot movement. This 
simulator is like any other in that it calculates all of the system dynamics in real time and from 
these computations come changes to the system states, which alter the visual scene observed by 
the physician. The visual scene is generated by high speed computer graphics engines not unlike 
those employed by modem flight simulators. However, a unique aspect of the proposed 
embodiment is that the graphics image is periodically updated by the video image transmitted 
over the long haul network. This approach ensures that the visual scenes at the patient and at the 
simulator are never allowed to deviate perceptibly. This update is generated by a complex 
scheme of image decoding, texture extraction and image format transformation. 

The intelligent controller performs the dual role of optimizing robot performance and preventing 
inadvertent incisions. The research will investigate two general approaches to the design. One 
approach will use optimal control theory and the other will utilize a hybrid of soft computing 
techniques (fuzzy control, neural networks and genetic algorithms). Both of these techniques 
have been used successfully to control autonomous aircraft. 

The simulator also calculates appropriate inceptor forces. In the near term, the drive signal math 
model for the haptic stimuli will be essentially the same as that in the actual robot although it 
will rely on a sophisticated organ dynamics model to compute the appropriate organ forces 
interacting with the robot end effectors. Eventually haptic feedback will be applied to enhance 
the environment for the surgeon. It has been shown in other applications that haptic stimuli, even 
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though artificial, provide information to the operator that improves human performance 
(reference Hannaford). The organ dynamics model will also provide organ state information to 
the simulator graphics generator. Another organ dynamics model will reside along with the 
intelligent controller in order to allow model reference control. 


9. Summary 

An extremely innovative approach has been presented, which is to have the surgeon operate 
through a simulator running in real-time enhanced with an intelligent controller component to 
enhance the safety and efficiency of a remotely conducted operation. The use of a simulator 
enables the surgeon to operate in a virtual environment free from the impediments of 
telecommunication delay. The simulator functions as a predictor and periodically the simulator 
state is corrected with “truth” data. 

Three major research areas must be explored in order to ensure achieving the objectives. They 
are: simulator as predictor, image processing, and intelligent control. Each is equally necessary 
for success of the project and each of these involves a significant intelligent component in it. 
These are diverse, interdisciplinary areas of investigation, thereby requiring a highly coordinated 
effort by all the members of our team, to ensure an integrated system. The following is a brief 
discussion of those areas. 

Simulator as a predictor: The delays encountered in remote robotic surgery will be greater than 
any encountered in human-machine systems analysis, with the possible exception of remote 
operations in space. Therefore, novel compensation techniques will be developed. Included will 
be the development of the real-time simulator, which is at the heart of our approach. The 
simulator will present real-time, stereoscopic images and artificial haptic stimuli to the surgeon. 

Image processing: Because of the delay and the possibility of insufficient bandwidth a high 
level of novel image processing is necessary. This image processing will include several 
innovative aspects, including image interpretation, video to graphical conversion, texture 
extraction, geometric processing, image compression and image generation at the surgeon 
station. 

Intelligent control: Since the approach we propose is in a sense “predictor” based, albeit a very 
sophisticated predictor, a controller, which not only optimizes end effector trajectory but also 
avoids error, is essential. We propose to investigate two different approaches to the controller 
design. One approach employs an optimal controller based on modern control theory; the other 
one involves soft computing techniques, i.e. fuzzy logic, neural networks, genetic algorithms and 
hybrids of these. 
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